{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a href=\"https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/examples/evaluation/Deepeval.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 🚀 DeepEval - Open Source Evals with Tracing\n",
    "\n",
    "This code tutorial shows how you can easily trace and evaluate your LlamaIndex Agents. You can read more about the DeepEval framework here: https://docs.confident-ai.com/docs/getting-started\n",
    "\n",
    "LlamaIndex integration with DeepEval allows you to trace your LlamaIndex Agents and evaluate them using DeepEval's default metrics. Read more about the integration here: https://deepeval.com/integrations/frameworks/langchain\n",
    "\n",
    "Feel free to check out our repository here on GitHub: https://github.com/confident-ai/deepeval"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Quickstart\n",
    "\n",
    "Install the following packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install -q -q llama-index\n",
    "!pip install -U -q deepeval"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This step is optional and only if you want a server-hosted dashboard! (Psst I think you should!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!deepeval login"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### End-to-End Evals\n",
    "\n",
    "`deepeval` allows you to evaluate LlamaIndex applications end-to-end in under a minute.\n",
    "\n",
    "Create a `FunctionAgent` with a list of metrics you wish to use, and pass it to your LlamaIndex application's `run` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import asyncio\n",
    "\n",
    "from llama_index.llms.openai import OpenAI\n",
    "import llama_index.core.instrumentation as instrument\n",
    "\n",
    "from deepeval.integrations.llama_index import (\n",
    "    instrument_llama_index,\n",
    "    FunctionAgent,\n",
    ")\n",
    "from deepeval.metrics import AnswerRelevancyMetric\n",
    "\n",
    "instrument_llama_index(instrument.get_dispatcher())\n",
    "\n",
    "\n",
    "def multiply(a: float, b: float) -> float:\n",
    "    \"\"\"Useful for multiplying two numbers.\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "answer_relevancy_metric = AnswerRelevancyMetric()\n",
    "\n",
    "agent = FunctionAgent(\n",
    "    tools=[multiply],\n",
    "    llm=OpenAI(model=\"gpt-4o-mini\"),\n",
    "    system_prompt=\"You are a helpful assistant that can perform calculations.\",\n",
    "    metrics=[answer_relevancy_metric],\n",
    ")\n",
    "\n",
    "\n",
    "async def llm_app(input: str):\n",
    "    return await agent.run(input)\n",
    "\n",
    "\n",
    "asyncio.run(llm_app(\"What is 2 * 3?\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluations are supported for LlamaIndex `FunctionAgent`, `ReActAgent` and `CodeActAgent`. Only metrics with LLM parameters input and output are eligible for evaluation.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Synchronous\n",
    "\n",
    "Create a `FunctionAgent` with a list of metrics you wish to use, and pass it to your LlamaIndex application's run method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepeval.dataset import EvaluationDataset, Golden\n",
    "\n",
    "dataset = EvaluationDataset(\n",
    "    goldens=[Golden(input=\"What is 3 * 12?\"), Golden(input=\"What is 4 * 13?\")]\n",
    ")\n",
    "\n",
    "for golden in dataset.evals_iterator():\n",
    "    task = asyncio.create_task(llm_app(golden.input))\n",
    "    dataset.evaluate(task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Asynchronous"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from deepeval.dataset import EvaluationDataset, Golden\n",
    "import asyncio\n",
    "\n",
    "dataset = EvaluationDataset(\n",
    "    goldens=[Golden(input=\"What's 7 * 8?\"), Golden(input=\"What's 7 * 6?\")]\n",
    ")\n",
    "\n",
    "for golden in dataset.evals_iterator():\n",
    "    task = asyncio.create_task(llm_app(golden.input))\n",
    "    dataset.evaluate(task)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ⚠️ Warning: DeepEval runs using event loops for managing asynchronous operations.\n",
    "\n",
    "Jupyter notebooks already maintain their own event loop, which may lead to unexpected behavior, hangs, or runtime errors when running DeepEval examples directly in a notebook cell.\n",
    "\n",
    "Recommendation: To avoid such issues, run your DeepEval examples in a standalone Python script (.py file) instead of within Jupyter Notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Examples\n",
    "\n",
    "Here are some examples scripts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Synchronous (End-to-End Evals)\n",
    "import os\n",
    "import deepeval\n",
    "import asyncio\n",
    "\n",
    "from llama_index.llms.openai import OpenAI\n",
    "import llama_index.core.instrumentation as instrument\n",
    "\n",
    "from deepeval.integrations.llama_index import instrument_llama_index\n",
    "from deepeval.integrations.llama_index import FunctionAgent\n",
    "from deepeval.metrics import AnswerRelevancyMetric\n",
    "from deepeval.dataset import EvaluationDataset, Golden\n",
    "\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "deepeval.login(os.getenv(\"CONFIDENT_API_KEY\"))\n",
    "instrument_llama_index(instrument.get_dispatcher())\n",
    "\n",
    "\n",
    "def multiply(a: float, b: float) -> float:\n",
    "    \"\"\"Useful for multiplying two numbers.\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "answer_relevancy_metric = AnswerRelevancyMetric()\n",
    "agent = FunctionAgent(\n",
    "    tools=[multiply],\n",
    "    llm=OpenAI(model=\"gpt-4o-mini\"),\n",
    "    system_prompt=\"You are a helpful assistant that can perform calculations.\",\n",
    "    metrics=[answer_relevancy_metric],\n",
    ")\n",
    "\n",
    "\n",
    "async def llm_app(input: str):\n",
    "    return await agent.run(input)\n",
    "\n",
    "\n",
    "dataset = EvaluationDataset(\n",
    "    goldens=[Golden(input=\"What is 3 * 12?\"), Golden(input=\"What is 4 * 13?\")]\n",
    ")\n",
    "for golden in dataset.evals_iterator():\n",
    "    task = asyncio.create_task(llm_app(golden.input))\n",
    "    dataset.evaluate(task)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Asynchronous (End-to-End Evals)\n",
    "import os\n",
    "from deepeval.integrations.llama_index import instrument_llama_index\n",
    "import llama_index.core.instrumentation as instrument\n",
    "from deepeval.integrations.llama_index import FunctionAgent\n",
    "from llama_index.llms.openai import OpenAI\n",
    "import asyncio\n",
    "import time\n",
    "\n",
    "import deepeval\n",
    "from deepeval.metrics import AnswerRelevancyMetric\n",
    "from deepeval.dataset import EvaluationDataset, Golden\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "\n",
    "# Don't forget to setup tracing\n",
    "deepeval.login(os.getenv(\"CONFIDENT_API_KEY\"))\n",
    "instrument_llama_index(instrument.get_dispatcher())\n",
    "\n",
    "\n",
    "def multiply(a: float, b: float) -> float:\n",
    "    \"\"\"Useful for multiplying two numbers.\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "answer_relevancy_metric = AnswerRelevancyMetric()\n",
    "agent = FunctionAgent(\n",
    "    tools=[multiply],\n",
    "    llm=OpenAI(model=\"gpt-4o-mini\"),\n",
    "    system_prompt=\"You are a helpful assistant that can perform calculations.\",\n",
    "    metrics=[answer_relevancy_metric],\n",
    ")\n",
    "\n",
    "goldens = [Golden(input=\"What's 7 * 8?\"), Golden(input=\"What's 7 * 6?\")]\n",
    "\n",
    "\n",
    "async def llm_app(golden: Golden):\n",
    "    await agent.run(golden.input)\n",
    "\n",
    "\n",
    "def main():\n",
    "    dataset = EvaluationDataset(goldens=goldens)\n",
    "    for golden in dataset.evals_iterator():\n",
    "        task = asyncio.create_task(llm_app(golden))\n",
    "        dataset.evaluate(task)\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "from deepeval.integrations.llama_index import instrument_llama_index\n",
    "import llama_index.core.instrumentation as instrument\n",
    "from deepeval.integrations.llama_index import FunctionAgent\n",
    "from llama_index.llms.openai import OpenAI\n",
    "import asyncio\n",
    "\n",
    "import deepeval\n",
    "from dotenv import load_dotenv\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "# Don't forget to setup tracing\n",
    "deepeval.login(os.getenv(\"CONFIDENT_API_KEY\"))\n",
    "instrument_llama_index(instrument.get_dispatcher())\n",
    "\n",
    "\n",
    "def multiply(a: float, b: float) -> float:\n",
    "    \"\"\"Useful for multiplying two numbers.\"\"\"\n",
    "    return a * b\n",
    "\n",
    "\n",
    "agent = FunctionAgent(\n",
    "    tools=[multiply],\n",
    "    llm=OpenAI(model=\"gpt-4o-mini\"),\n",
    "    system_prompt=\"You are a helpful assistant that can perform calculations.\",\n",
    "    metric_collection=\"test_collection_1\",\n",
    ")\n",
    "\n",
    "\n",
    "async def llm_app(golden: Golden):\n",
    "    await agent.run(golden.input)\n",
    "\n",
    "\n",
    "asyncio.run(llm_app(Golden(input=\"What is 3 * 12?\")))"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
